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ABSTRACT 


This paper introduces the concept of a responsive environ- 
ment which perceives human behavior and responds with 
intelligent auditory and visual feedback. Several exhibits of 
responsive environments, implemented by the author, com- 
bining computer graphics, video projection and two-way 
video communication are described. VIDEOPLACE, an 
evolving exhibit which defines a conceptual telecommuni- 
cation environment uniting geographically separated people 
in a common visual experience, is discussed at some length. 
Based on these examples a new art form of composed man- 
machine interaction is defined. Finally, practical applica- 
tions are suggested for the fields of education, psychology 
and psychotherapy. 


INTRODUCTION 


Man-machine interaction is usually limited to a seated man 
poking at a machine with his fingers or perhaps waving a 
wand over a data tablet. Seven years ago, I was dissatisfied 
with such a restricted dialogue and embarked on research 
exploring more interesting ways for men and machines to 
relate. The result was the concept of a responsive environ- 
ment in which a computer perceives the actions of those 
who enter and responds intelligently through complex vis- 
ual and auditory displays. 

Over a period of time the computer’s displays establish a 
context within which the interaction occurs. It is within this 
context that the participant chooses his next action and 
anticipates the environment’s response. If the response is 
unexpected, the environment has changed the context and 
the participant must reexamine his expectations. The expe- 
rience is controlled by a composition which anticipates the 
participant’s actions and flirts with his expectations. 

This paper describes the evolution of these concepts from 
their primitive beginnings to my current project, VIDEO- 
PLACE, which provides a general tool for devising many 
interactions. Based on these examples an interactive art 
form is defined and its promise identified. While the envi- 
ronments described were presented with aesthetic intent. their 
implications go beyond art. In the final section, applications 
in education, psychology and psychotherapy are suggested. 


4 


GLOWFLOW 


In 1969, I became involved in the development of 
GLOWFLOW, a computer art project conceived by Dan 
Sandin, Jerry Erdman and Richard Venezsky at the Univer- 
sity of Wisconsin. It was designed in an atmosphere of 
encounter between art and technology. The viewer entered 
a darkened room in which glowing lines of light defined an 
illusory space (Figure 1). The display was accomplished by 
pumping phosphorescent particles through transparent 
tubes attached to the gallery walls. These tubes passed 


through opaque columns concealing lights which excited 
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the phosphors. A pressure sensitive pad in front of each of 
the six columns enabled the computer to respond to foot- 
steps by lighting different tubes or changing the sounds 
generated by a Moog synthesizer or the origin of these 
sounds. However, the artists’ attitude toward the capacity 
for response was ambivalent. They felt that it was impor- 
tant that the environment respond, but not that the audi- 
ence be aware of it. Delays were introduced between the 
detection of a participant and the computer’s response so 
that the contemplative mood of the environment would not 
be destroyed by frantic attempts to elicit more responses. 

While GLOWFLOW was quite successful visually, it 
succeeded more as a kinetic sculpture than as a responsive 
environment. However, the GLOWFLOW experience led 
me tea mimber af decisions: 


1. Interactive art is potentially a richly composable me- 
dium quite distinct from the concerns of sculpture, 
graphic art or music. 

2. In order to respond intelligently the computer should 
perceive as much as possible about the participant’s 
behavior. 

3. In order to focus on the relationships between the 
environment and the participants, rather than among 
participants, only a small number of people should be 
involved at a time. 

4. The participants should be aware of how the environ- 
ment is responding to them. 

5. The choice of sound and visual response systems 
should be dictated by their ability to convey a wide 
variety of conceptual relationships. 

6. The visual responses should not be judged as art nor 
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Figure 1—Glowflow tubes on gallery wall 


the sounds as music. The only aesthetic concern is the 
quality of the interaction. 


METAPLAY 


Following the GLOWFLOW experience, I conceived and 
directed METAPLAY which was exhibited in the Memorial 
Union Gallery of the University of Wisconsin for a month 
in 1970. It was supported by the National Science Founda- 
tion, the Computer Science Department, the Graduate 
School and the loan of a PDP-12 by Digital Equipment 
Corporation. 

METAPLAY’S focus reflected my reactions to GLOW- 
FLOW. Interaction between the participants and the envi- 
ronment was emphasized; the computer was used to facili- 
tate a unique real-time relationship between the artist and 
the participant. An 8’ by 10’ rear-projection video screen 
dominated the gallery. The live video image of the viewer 
and a computer graphic image drawn by an artist, who was 
in another building, were superimposed on this screen. 
Both the viewer and the artist could respond to the result- 
ing image. 


Hardware 


The image communications (Figure 2) started with an 
analogue data tablet which enabled the artist to draw or 
write on the computer screen. The person doing the draw- 
ing did not have to be an artist, but the term is used for 
convenience. One video camera, in the Computer Center, 
was aimed at the display screen of the Adage Graphic 
Dispiay Computer. A second camera, a miie away in the 
gallery, picked up the live image of people in the room. A 
television cable transmitted the video computer image from 
the Computer Center to the gallery and the two signals 
were mixed so that the computer image overlayed the live 
image. The composite image was projected on the 8’ x 10’ 
screen in the gallery and was simultaneously transmitted 
back to the Computer Center where it was displayed on a 
video monitor providing feedback for the artist. 


The artist could draw on the Adage screen using a data 
tablet. By using function switches, potentiometers and the 
teletype keyboard the pictures could be rapidly modified or 
the mode of drawing itself altered. In addition to the effects 
of simple drawings, the image could be moved around the 
screen, image size could be controlled and the picture could 
be repeated up to ten times on the screen displaced by 
variable X, Y and size increments. A tail of a fixed number 
of line segments could be drawn allowing the removal of a 
segment at one end while another was added at the opposite 
end. An image could be rotated in 3-space under control of 
the pen. Although this was not true rotation, the visual 
effect was similar. A simple set of transformations under 
potentiometer and tablet control yielded apparent animation 
of people’s outlines. Finally, previously defined images 
could be recalled or exploded. While it might seem that the 
drawing could be done without a computer, the ability to 
rapidly erase, recall and transform images required consid- 
erable processing and created a far more powerful means of 
expression than pencil and paper could provide. 


Interaction 


These facilities provided a rich repertoire for an unusual 
dialogue. The artist could draw pictures on the participants’ 
images or communicate directly by writing words on the 
screen (Figure 3). He could induce people to play a game 
like Tic-Tac-Toe or play with the act of drawing, starting to 
draw one kind of picture only to have it transformed into 
another by interpolation. 


Live graffiti 


One interaction derived from the artist’s ability to draw 
on the image of the audience. He could add graffiti-like 
features or animate a drawn outline of a person so that it 
appeared to dance to the music in the gallery. The artist 
tried various approaches to involve people in the interac- 
tion. Failing to engage one person, he would seek someone 
more responsive. 

It was important to involve the participants in the act of 
drawing. However, the electronic wand designed for this 
purpose did not work reliably. What evolved was a serendi- 
pitous solution. One day as I was trying to draw on a 
student’s hand, he became confused and moved it. When I 
erased my scribblings and started over, he moved his hand 
again. He did this repeatedly until it became a game. 
Finally, it degenerated to the point where I was simply 
tracking the image of his hand with the computer line. In 
effect, by moving his hand he could draw on the screen 
before him. 

The relationship established with this participant was 
developed as one of the major themes of METAPLAY. It 
was repeated and varied until it became an aesthetic 
medium in iteslf. With each person we involved in this way, 
we tried to preserve the pleasure of the original discovery. 
After playing some graffiti games with each group that 


ta 
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Figure 2—Metaplay communications 


entered, we would focus on a single individual and draw 
around the image of his hand. After an initial reaction of 
blank bewilderment, the self-conscious person would make 
a nervous gesture. The computer line traced the gesture. A 
second gesture, followed by the line was the key to 
discovery. One could draw on the video screen with his 
finger! Others in the group, observing this phenomenon, 
would want to try it too. The line could be passed from one 
person’s finger to another’s. Literally hundreds of interac- 
tive vignettes developed within this simple communication 
channel. 

Drawing by this method was a rough process. Pictures of 
any but the simplest shapes were unattainable. This was 
mainly because of the difficulty of tracking a person’s 


finger. Happily, neither the artist nor the audience were 
concerned about the quality of the drawings. What was 
exciting was interacting in this novel way through a man- 
computer-video link spanning a mile. 


PSYCHIC SPACE 


The next step in the evolution of the responsive environ- 
ment was PSYCHIC SPACE, which I designed and exhib- 
ited in the Memorial Union Gallery during May and June of 
1971. It was implemented with the help of my students, the 
Computer Science Department and a National Science 
Foundation grant in Complex Information Processing. 
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Figure 3—Metaplay drawing 


PSYCHIC SPACE was both an instrument for musical 
expression and a richly composed, interactive, visual expe- 
rience. Participants could become involved in a softshoe 
duet with the environment, or they could attempt to match 
wits with the computer by walking an unpredictable maze 
projected on an 8’ X 10’ video screen. 


Hardware 


A PDP-11 had direct control of all sensing and sound in 
the gallery. In addition, it communicated with the Adage 
AGT-10 Graphic Display Computer at the Computer Center 
(Figure 4). The Adage image was transmitted over video 
cable to the gallery where it was rear-projected on the 8’ x 
10’ screen. The participant’s position on the floor was the 
basis for each of the interactions. The sensing was done by 
a 16’ X 24’ grid of pressure switches, constructed in 2’ x 4’ 
modules, each containing eight switches (Figure 5). Since 
they were electronically independent, the system was able 
to discriminate among individuals if several were present. 
This independence made it easy for the programming to 
ignore a faulty switch until its module was replaced or 
repaired. Since there were 16 bits in the input words of the 
PDP-11, it was natural to read the 16 switches in each row 
across the rcom in parallel (Figure 6). Digital circuitry wa 
then used to scan the 24 rows under computer control. 
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Figure 4—Data and video communication for psychic space 


Input and interaction 


Since the goal was to encourage the participants to 
express themselves through the environment, the program 
automatically responded to the footsteps of people entering 
the room with electronic sound. We experimented with a 
number of different schemes for actually generating the 
sounds based on an analysis of peoples’ footsteps. In 
sampling the floor 60 times per second we discovered that a 
single footstep consisted of as many as four discrete 
events: lifting the heel, lifting the toe, putting the heel down 
and putting the ball of the foot down. The first two were 
dubbed the ‘‘unfootstep.”’ We could respond to each foot- 
step or unfootstep as it occurred, or we could respond to 
the person’s average position. A number of response 
schemes were tried, but the most pleasing was to start each 








Figure S—Flooring sensing modules in psychic space 
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im creak 
Figure 6—Participants’ feet are seen by the computer as ones in a field of 
zeroes 


tone only when a new switch was stepped on and then to 
terminate it on the next ‘‘unfootstep.’’ Thus it was possible 
to get silence by jumping, or by lifting one foot, or by 
putting both feet on the same switch. 

__ Typical reaction to the sounds was instant understanding, 
followed by a rapid-fire sequence of steps, yumps and roils. 
This phase was followed by a slower more thoughtful 
exploration of the environment in which more subtle and 
interesting relationships could be developed. In the second 
phase, the participant would discover that the room was 
organized with high notes at one end and low notes at the 
other. After a while, the keyboard was abruptly rotated by 
90 degrees. 

After a longer period of time an additional feature came 
into play. If the computer discovered that a person’s 
behavior was characterized by a short series of steps 
punctuated by relatively long pauses, it would use the 
pause to establish a new kind of relationship. The sequence 
of steps was responded to with a series of notes as before; 
however, during the pause the computer would repeat these 
notes again. If the person remained still during the pause, 
the computer assumed that the relationship was under- 
stood. The next sequence of steps was echoed at a noticea- 


bly higher pitch. Subsequent sequences were repeated. 
several times with variations each time. This interaction 
was experimentai and extremely difficult to introduce 
clearly with feedback alone, i.e., without explicit instruc- 
tions. The desire was for a man-machine dialogue resem- 
bling the guitar duel in the film ‘‘Deliverance.”’ 


MAZE—A COMPOSED ENVIRONMENT 


The maze program focused on the interaction between 
one individual and the environment. The participant was 
lured into attempting to navigate a projected maze. The 
intrigue derived from the maze’s responses, a carefully 
composed sequence of relations designed to constitute a 
unique and coherent experience. 


Hardware 


The maze itself was not programmed on the PDP-11, but 
on the Adage located a mile away in the Computer Center. 
The PDP-11 transmitted the participant’s floor coordinates 
across an audio cable to the Adage. The data was transmit- 
ted asynchronously as a serial bit stream of varying pulse 
widths. The Adage generated the maze image which was 
picked up by a TV camera and transmitted via a video cable 
back to the Union where it was rear-screen projected to a 
size of 8’ x 10’. 


Interaction 


The first problem was simply to educate the person to the 
relationships between the floor and the screen. Initially, a 
diamond with a cross in it representing the person’s posi- 
tion appeared on the screen. Physical movement in the 
room caused the symbol to move correspondingly on the 
screen. As the participant approached the screen, the 
symbol moved up. As he moved away, it moved down. The 
next step was to induce the person to move to the starting 
point..of the. maze, which had not yet appeared on. the 
screen (Figure 7). To this end, another object was placed on 
the screen at the position which would be the starting point 
of the maze. The viewer unavoidably wondered what would 
happen if he walked his symbol to the object. The arrival of 
his symbol at the starting point caused the object to vanish 
and the maze to appear. Thus confronted with the maze, no 
one questioned the inevitability of walking it. 


Software boundaries 


Since there was no physical constraints in the gallery, the 
boundaries of the maze had to be enforced by the com- 
puter. Each attempt to violate a boundary was foiled by one 
of many responses in the computer’s repertoire. The com- 
puter could move the line, stretch it elastically, or move the 
whole maze. The line could disappear, seemingly removing 
the barrier, except that the rest of the maze would change 
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Figure 7—Composed environment-Maze 


simultaneously so no advantage was gained. In addition, 
the symbol representing the person could split in half at the 
violated boundary, with one half held stationary while the 
other half, the alter ego, continued to track movement. 
However, no progress could be made until the halves of the 
symbol were reunited at the violated boundary. 

Even when the participant was moving legally, there 
were changes in the program contingent upon his position. 
Several times, as the goal was approached, the maze 
changed to thwart immediate success. Or, the relationship 
between the floor and the maze was altered so that move- 
ments that once resulted in vertical motion, now resulted in 
horizontal motion. Alternatively, the symbol representing 
the participant could remain stationary while the maze 
moved. 

Ultimately, success was not allowed. When reaching the 
goal seemed imminent, additional boundaries appeared in 
front of and behind the symbol, boxing it in. At this point, 
the maze slowly shrank to nothing. While the goal could not 
be reached, the composed frustration made the route inter- 
esting. 


Experience 


The maze experience conveyed a unique set of feelings. 
The video display space created a sense of detachment 


enhanced by the displaced feedback; movement on the 
horizontal plane of the floor translated onto the vertical 
plane of the screen. The popular stereotype of dehumaniz- 
ing technology seemed fulfilled. However, the maze idea 
was engaging and people became involved willingly. The 
lack of any other sensation focused attention completely on 
this interaction. As the experience progressed, their per- 
ception of the maze changed. From the initial impression 
that it was a problem to solve, they moved to the realiza- 
tion that the maze was a vehicle for whimsy, playing with 
the concept of a maze and poking fun at their compulsion to 
walk it. 


VIDEOPLACE 


For the past two years I have been working on a project 
called VIDEOPLACE, under the aegis of the Space Sci- 
ence and Engineering Center of the University of Wiscon- 
sin. This work is funded by the National Endowment for 
the Arts and the Wisconsin Arts Board. A preliminary 
version was exhibited at the Milwaukee Art Center for six 
weeks beginning in October 1975. The development of 
VIDEOPLACE is still under way and several more years 
will be required before its potential is fully realized both in 
terms of implementing the ore hardware and exploring 
its compositional possibilities 





VIDEOPLACE is a conceptual environment with no 
physical existence. It unites people in separate locations in 
a common visual experience, allowing them to interact in 
unexpected ways through the video medium. The term 
VIDEOPLACE is based on the premise that the act of 
communication creates a place that consists of all the 
information that the participants share at that moment. 
When people are in the same room, the physical and 
communication places are the same. When the communi- 
cants are separated by distance, as in a telephone conversa- 
tion, there is still a sense of being together although sight 
and touch are not possible. By using television instead of 
telephone, VIDEOPLACE seeks to augment this sense of 
place by including vision, physical dimension and a new 
interpretation of touch. 

VIDEOPLACE consists of two or more identical envi- 
ronments which can be adjacent or hundreds of miles apart. 
In each environment , a single person walks into a darkened 
room where he finds himself confronted by an 8’ x 10’ rear- 
view projection screen. On the screen he sees his own life- 
size image and the image of one or more other people. This 
is surprising in itself, since he is alone in the room (Figure 
8). The other images are of people in the other environ- 
ments. They see the same composite image on their 
screens. The visual effect is of several people in the same 
room. By moving around their respective rooms, thus 
moving their images, the participants can interact within the 
limitations of the video medium. 

It is these apparent limitations that I am currently work- 
ing to overcome. When people are physically together, they 





Figure 8—Videoplace 


can talk, move around the same space, manipulate the same 
objects and touch each other. All of these actions would 
appear to be impossible within the VIDEOPLACE. How- 
ever, the opposite is true. The video medium has the 
potential of being more rich and variable in some ways, 
than reality itself. 

It would be easy to allow the participants to talk, 
although I usually preclude this, to force people to focus on 
the less familiar kinds of interaction that the video medium 
provides. A sense of dimension can be created with the 
help of computer graphics, which can define a room or 
another spatial context within which the participants appear 
to move around. Graphics can also furnish this space with 
artificial objects and inhabit it with imaginary organisms. 
The sense of touch would seem to be impossible to dupli- 
cate. However, since the cameras see each person’s image 
in contrast to a neutral background, it is easy to digitize the 
outline and to determine its orientation on the screen 
(Figure 9). It is also easy to tell if one person’s image 
touches another’s, or if someone touches a computer 
graphic object. Given this information the computer can 
make the sense of touch effective. It can currently respond 
with sounds when two images touch and will ultimately 
allow a person’s image to pick up a graphic object and 
move it about the screen. 

While the participants’ bodies are bound by physical laws 
such as gravity, their images could be moved around the 
screen, shrunk, rotated, colorized and keyed together in 
arbitrary ways. Thus, the full power of video processing 





Figure 9—The video outline sensor 
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could be used to mediate the interaction and the usual laws 
of cause and effect replaced with alternatives composed by 
the artist. 

The impact of the experience will derive from the fact 
that each person has a very proprietary feeling towards his 
own image. What happens to his image happens to him. In 
fact, when one person’s image overlaps another’s, there is a 
psychological sensation akin to touch. In VIDEOPLACE, 
this sensation can be enhanced in a number of ways. One 
image can occlude the other. Both images can disappear 
where they intersect. Both images can disappear except 
where they intersect. The intersection of two images can be 
used to form a window into another scene so two partici- 
pants have to cooperate to see a third. 

VIDEOPLACE need not involve more than one partici- 
pant. It is quite possible to create a compelling experience 
for one person by projecting him into this imaginary domain 
alone. In fact the hardware/software system underlying 
VIDEOPLACE is not conceived as a single work but as a 
general facility for exploring all the possibilities of the 
medium to be described next. 


RESPONSE IS THE MEDIUM 


The environments described suggest a new art medium 
based on a commitment to real-time interaction between 
men and machines. The medium is comprised of sensing, 
display and control systems. It accepts inputs from or about 
the participant and then outputs in a way he can recognize 
as corresponding to his behavior. The relationship between 
inputs and outputs is arbitrary and variable, allowing the 
artist to intervene between the participant’s action and the 
results perceived. Thus, for example, the participant’s 
physical movement can cause sounds or his voice can be 
used to navigate a computer defined visual space. It is the 
composition of these relationships between action and 
response that is important. The beauty of the visual and 
aural response is secondary. Response is the medium! 

The distinguishing aspect of the medium is, of course, the 
fact that it responds to the viewer in an interesting way. In 
order to do this, it must know as much as possible about 
what the participant is doing. It cannot respond intelligently 
if it is unable to distinguish various kinds of behavior as 
they occur. 

The environment might be able to respond to the partici- 
pant’s position, voice volume or pitch, position relative to 
prior position or the time elapsed since the last movement. 
It could also respond to every third movement, the rate of 
movement, posture, height, colors of clothing or time 
elapsed since the person entered the room. If there were 
several people in the room, it might respond to the distance 
separating them, the average of their positions or the 
computer’s ability to resolve them, i.e., respond differently 
when they are very close together. 

In more complex interactions like the maze, the com- 
puter can create a context within which the interaction 
occurs. This context is an artificial reality within which the 
artist has complete control of the laws of cause and effect. 


Thus the actions perceived by the hardware sensors are 
tested for significance within the current context. The 
computer asks if the person has crossed the boundary in the 
maze or has touched the image of a particular object. At a 
higher level the machine can Jearn about the individual and 
judge from its past experience with similar individuals just 
which responses would be most effective. 

Currently, these systems are constrained by the total 
inability of the computer to make certain very useful and 
for the human, very simple perceptual judgments, such as 
whether a given individual is a man or a woman or is young 
or old. The perceptual system will define the limits of 
meaningful interaction, for the environment cannot respond 
to what it cannot perceive. To date the sensing systems 
have included pressure pads, ultrasonics and video digitiz- 
ing. 

As mentioned before, the actual means of output are not 
as important in this medium as they would be if the form 
were conceived as solely visual or auditory. In fact, it may 
be desirable that the output not qualify as beautiful in any 
sense, for that would distract from the central theme: the 
relationship established between the observer and the envi- 
ronment. Artists are fully capable of producing effective 
displays in a number of media. This fact is well known and 
to duplicate it produces nothing new. What is not known 
and remains to be tested is the validity of a responsive 
aesthetic. 

It is necessary that the output media be capable of 
displaying intelligent, or at least composed reactions, so 
that the participant knows which of his actions provoked it 
and what the relationship of the response is to his action. 
The purpose of the displays is to communicate the relation- 
ships that the environment is trying to establish. They must 
be capable of great variation and fine control. The response 
can be expressed in light, sound mechanical movement, or 
through any means that can be perceived. So far computer 
graphics, video generators, light arrays and sound synthe- 
sizers have been used. 


CONTROL AND COMPOSITION 


The control system includes hardware and software con- 
trol of all inputs and outputs as well as processing for 
decisions that are programmed by the artist. He must 
balance his desire for interesting relationships against the 
commitment to respond in real-time. The simplest re- 
sponses are little more than direct feedback of the partici- 
pant’s behavior, allowing the environment to show off its 
perceptual system. But far more sophisticated results are 
possible. In fact, a given aggregation of hardware sensors, 
displays and processors can be viewed as an instrument 
which can be programmed by artists with differing sensitivi- 
ties to create completely different experiences. The envi- 
ronment can be thought of in the following ways: 


1. An entity which engages the participant in a dialogue. 
The environment expresses itself through light and 


sound while the participant communicates with physi- 
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cal motion. Since the experience is an encounter 
between individuals, it might legitimately include 
greetings, introductions and farewells—all in an ab- 
stract rather than literal way. The problem is to 
provide an interesting personality for the environment. 

2. A personal amplifier. One individual uses the environ- 
ment to enhance his ability to interact with those 
within it. To the participants the interaction might 
appear similar to that described above. The result 
would be limited by the speed of the artist’s response 
but improved by his sensitivity to the participants’ 
moods. The live drawing interaction in METAPLAY 
could be considered an example of this approach. 

3. An environment which has sub-environments with 
different response relationships. This space could be 
inhabited by artificial organisms defined either visually 
or with sound. These creatures can interact with the 
participants as they move about the room. 

4. An amplifier of physical position in a real or artificially 
generated space. Movements around the environment 
would result in much larger apparent movements in 
the visually represented space. A graphic display 
computer can be used to generate a perspective view 
of a modelled space as it would appear if the partici- 
pant were within it. Movements in the room would 
result in changes in the display, so that by moving 
only five feet within the environment, the participant 
would appear to have moved fifty feet in the display. 
The rules of the modelled space can be totally arbi- 
trary and physically impossible, e.g. a space where 
objects recede when you approach them. 

5. An instrument which the participants play by moving 
about the space. In PSYCHIC SPACE the floor was 
used as a keyboard for a simple musical instrument. 

6. A means of turning the participant’s body into an 
instrument. His physical posture would be determined 
from a digitized video image and the orientation of the 
limbs would be used to control lights and sounds. 

7. A game between the computer and the participant. 
This variation is really a far more involving extension 
of the pinball machine, already the most commercially 
‘succes sfil interactive environment. 

8. An experimental parable where the theme is illustrated 
by the things that happen to the protagonist—the 
participant. Viewed from this perspective, the maze in 
PSYCHIC SPACE becomes pregnant with meaning. It 
was impossible to succeed, to solve the maze. This 
could be a frustrating experience if one were trying to 
reach the goal. If, on the other hand, the participant 
maintained an active curiosity about how the maze 
would thwart him next, the experience was entertain- 
ing. Such poetic composition of experience is one of 
the most promising lines of development to be pursued 
with the environments. 


IMPLICATIONS OF THE ART FORM 


For the artist the environment augurs new relationships 
with his audience and his art. He operates at a metalevel. 


The participant provides the direct performance of the 
experience. The environmental hardware is the instrument. 
The computer acts much as an orchestra conductor control- 
ling the broad relationships while the artist provides the 
score to which both performer and conductor are bound. 
This relationship may be a familiar one for the musical 
composer, although even he is accustomed to being able to 
recognize one of his pieces, no matter who is interpreting it. 
But the artist’s responsibilities here become even broader 
than those of a composer who typically defines a detailed 
sequence of events. He is composing a sequence of possi- 
bilities, many of which will not be realized for any given 
participant who fails to take the particular path along which 
they lie. 

Since the artist is not dedicated to the idea that his entire 
piece be experienced he can deal with contingencies. He 
can try different approaches, different ways of trying to 
elicit participation. He can take into account the differences 
among people. In the past, art has often been a one-shot, 
hit-or-miss proposition. A painting could accept any atten- 
tion paid it, but could do little to maintain interest once it 
had started to wane. In an environment the loss of attention 
can be sensed as a person walks away. The medium can try 
to regain attention and upon failure, try again. The piece 
has a second strike capability. In fact it can learn to 
improve its performance, responding not only to the mo- 
ment but also to the entire history of its experience. 

In the environment, the participant is confronted with a 
completely new kind of experience. He is stripped of his 
informed expectations and forced to deal with the moment 
in its own terms. He is actively involved, discovering that 
his limbs have been given new meaning and that he can 
express himself in new ways. He does not simply admire 
the work of the artist; he shares in its creation. The 
experience he achieves will be unique to his movements 
and may go beyond the intentions of the artist or his 
understanding of the possibilities of the piece. 

Finally, in an exciting and frightening way, the environ- 
ments dramatize the extent to which we are savages in a 
world of our own creation. The layman has extremely little 
ability to define the limits of what is possible with current 
technology and so will accept all Sorts of cuies as represent- 
ing relationships which in fact do not exist. The constant 
birth of such superstitions indicates how much we have 
already accomplished in mastering our natural environment 
and how difficult the initial discoveries must have been. 


APPLICATIONS 


The responsive environment is not limited to aesthetic 
expression. It is a potent tool with applications in many 
fields. VIDEOPLACE clearly generalizes the act of tele- 
communication. It creates a form of communication so 
powerful that two people might choose to meet visually, 
even if it were possible for them to meet physically. While 
it is not immediately obvious that VIDEOPLACE is the 
optimum means of telecommunication, it is reasonably fair 
to say that it provides an infinitely richer interaction than 
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Picturephone allows. It broadens the range of possibilities 
beyond current efforts at teleconferencing. Even in its fetal 
stage, VIDEOPLACE is far more flexible than the tele- 
phone is after one hundred years of development. At a time 
when the cost of transportation is increasing and fiber 
optics promise to reduce the cost of communication, it 
seems appropriate to research the act of communication in 
an intuitive sense as well as in the strictly scientific and 
problem-solving approaches that prevail today. 


EDUCATION 


Responsive environments have tremendous potential for 
education. Our entire educational system is based on the 
assumption that thirty children will sit still in the same 
room for six hours a day and learn. This phenomenon has 
never been observed in nature and it’s the exception in the 
classroom where teachers are pitted against children’s 
natural desire to be active. The responsive environments 
offer a learning situation in which physical activity is 
encouraged. It is part of the process. An environment like 
VIDEOPLACE has an additional advantage. It gives the 
child a life-size physically identical alter ego who takes part 
in composed learning adventures on the video screen. In a 
fully developed VIDEOPLACE the size and position of the 
child’s image on the screen would be independent of actual 
location in the room. In an interactive Sesame Street a child 
would be mesmerized as his own miniaturized image was 
picked up by a giant Big Bird (Figure 10). Conversely he 
would be delighted if the scales were reversed and he were 
able to pick up the image of a tiny adult teacher who spoke 
to him from his hand. The most overworked educational 
cliché, ‘‘experience is the best teacher,’’ would have new 
meaning in this context. The environments provide an 
exerience which can be composed and condensed to dem- 
onstrate an educational point. 





Figure !0—Interactive sesame street 


While it is easy to generate examples of how the environ- 
ments can be used to teach traditional subjects, their 
significance does not lie only in their ability to automate 
traditional teaching. More important, they may revolution- 
ize what we teach as well as how we teach. Since the 
environments can define interesting relationships and 
change them in complex ways, it should be possible to 
create interactions which enrich the child’s conceptual 
experience. This would provide the child with more power- 
ful intellectual structures within which to organize the 
specific information he will acquire later. The goal would be 
to sophisticate the child, not to feed him facts. 


PSYCHOLOGY 


Since the environments can monitor the participants’ 
actions and respond with visual and auditory feedback, it is 
natural to consider their application to the study of human 
behavior. The use of the computer allows an experimenter 
to generate patterns and rhythms of stimuli and reinforcers. 
In addition, the ability to deal with gross physical behavior 
would suggest new experimental directions. For instance, 
perception could be studied as part of physical behavior 
and not as a sedentary activity distinct from it. Also, an 
environment like VIDEOPLACE is very general. The same 
aggregate of hardware and software could be programmed 
to control a broad range of experiments. The scheduling of 
different experiments could be interspersed because only 
the software would have to be changed. 

Since the university students used as subjects in many 
experiments are quite sophisticated about the concerns of 
psychologists, what is often being studied is the self- 
conscious behavior of people who Know they are in an 
experiment and are trying to second-guess it. On the other 
hand, environments open to the public offer a source of 
spontaneous behavior. It is quite easy for the computer to 
take statistics without interfering with the experience. Or, 
interactions can be composed to test specific experimental 
hypotheses. 


PSYCHOTHERAPY 


It is also worth considering the application of responsive 
environments to psychotherapy. Perhaps most important 
for a psychotherapist is the ability of the environment to 
evoke and expand behavior. We have found in the past that 
people alone in a dark room often become very playful and 
flamboyant—far more so than they are in almost any other 
situation. Since the environment is kept dark, the patient 
has a sense of anonymity; he can do things that he might 
not do otherwise. The fact that he is alone in the dark 
serves to protect him both from his image of himself and 
from his fear of other people. The darkness also is a form of 
sensory deprivation which might prevent a patient from 
withdrawing. If he is to receive any stimulation at all, it 


must be from acting within the environment. Once he acts 
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he can be reinforced for continuing to act. 





In the event that the subject refuses to act, the environ- 
ment can focus on motions so small as to be unavoidable 
and respond to these and as time goes by encourage them, 
slowly expanding them into larger behavior, ultimately 
leading the patient to extreme or cathartic action. 

In certain situations the therapist essentially programs 
himself to become mechanical and predictable, providing a 
structure that the patient can accept which can be expanded 
slowly beyond the original contract. It is possible that it 
would be easier to get a patient to trust a mechanical 
environment and completely mechanized therapy. Once the 
patient was acting and trusting within the environment, it 
would be possible to slowly phase in some elements of 
change, to generalize his confidence. As time went by, 
human images and finally human beings might be added. At 
this point, the patient could venture from his responsive 
womb, returning to it as often as needed. 


CONCLUSION 


The responsive environment has been presented as the 
basis for a new aesthetic medium based on real-time 
interaction between men and machines. In the long range it 
augurs a new realm of human experience, artificial realities 
which seek not to simulate the physical world but to define 


arbitrary, abstract and otherwise impossible relationships 
between action and result. In addition, it has been sug- 
gested that the concepts and tools of the responsive envi- 
ronments can be fruitfully applied in a number of fields. 

What perhaps has been obscured is that these concepts 
are the result of a personal need to understand and express 
the essence of the computer in humanistic terms. An earlier 
project to teach people how to use the computer was 
abandoned in favor of exhibits which taught people about 
the computer by letting them experience it. METAPLAY, 
PSYCHIC SPACE and VIDEOPLACE were designed to 
communicate an affirmative vision of technology to the lay 
public. This level of education is important, for our culture 
cannot continue if a large proportion of our population is 
hostile to the tools that define it. 

We are incredibly attuned to the idea that the sole 
purpose of our technology is to solve problems. It also 
creates concepts and philosophy. We must more fully 
explore these aspects of our inventions, because the next 
generation of technology will speak to us, understand us, 
and perceive our behavior. It will enter every home and 
office and intercede between us and much of the informa- 
tion and experience we receive. The design of such intimate 
technology is an aesthetic issue as much as an engineering 
one. We must recognize this if we are to understand and 
choose what we become as a result of what we have made. 


